Speed up workspace compilation in Eclipse by avoiding unnecessary rebuild cascades - #7209
Speed up workspace compilation in Eclipse by avoiding unnecessary rebuild cascades#7209chrisrueger with Copilot wants to merge 1 commit into
Conversation
|
First experiments in my larger project look very promising. Simple java changes which do not change public API or the manifest do not cause a downstream rebuild cascade. Only e.g. changes to interfaces , method signatures etc. would cause a downstream rebuild of dependant projects like before. This makes development nicer since it cuts down waiting time a lot. |
|
@peterkir FYI Let's discuss next week. |
bb790c6 to
8ec68dc
Compare
b33fb78 to
3b3afe9
Compare
|
@peterkir I now added a status indicator in the Toolbar as we discussed some weeks ago. So let's discuss this what we do with the feature once I'm back. BTW: I am using the feature already in my local bndtools which I used for developing the last part of this PR :) |
c450c92 to
1c3f985
Compare
|
This is a promising development. I just wanted to remind us collectively of #3175 as well for consideration. It had a similar aim but which would also provide some additional benefits. In summary: Eclipse's built-in project dependency checking already does this kind of more granular dependency checking, but only with project dependencies - jar dependencies are treated coarsely. More precisely: if you have a project dependency A, and a class file changes in project A that your project B doesn't use directly, then it will not rebuild your project B. This also applies (from memory) if (eg) you change a comment in the source file - the generated class file doesn't change, so your dependent project will not rebuild. With a jar dependency, if the timestamp on the jar is updated, then it will rebuild your project even if none of the classes in the jar have actually changed. A Bndtools workspace basically forces the coarse dependency behaviour because BndContainer adds every project dependency as a project and as a jar. We do this because Bnd can generate jars in complex ways that pull in other classes that are not in the project and so it is gives greater fidelity; however my initial investigations found that Eclipse should support emulating the same dependency behaviour to the same level of fidelity using dependency rules and allowing a project to propagate transitive dependencies. Fixing it this way would also fix a number of other problems that are caused by having both project and jar on the classpath (eg, source lookup F3 sometimes takes you to the Jar rather than original source file; duplicate search results, etc). It's been a long time, but last time I looked at it I nearly had it working but I think there was a bug somewhere in Eclipse that was preventing the dependency propagation from working as advertised, and so I had to abandon it. I was also trying to build a regression test suite in preparation for this change, which again was almost working but I was having niggling flakiness caused by race conditions. |
|
Thanks @kriegfrj for your comment. For completeness I want to emphasize that the approach in this PR is purely inside bnd core and not IDE/Eclipse specific. The part which touches bndtools (Eclipse) is mainly for display purposes and to configure the behaviour from bndtools (Eclipse bndtools preferences).
This "timestamp of the jar" is the main idea of this PR. In and I detect the kind of change of the bundle (was it an API change or change affecting consumers or purely internal) But of course maybe it could be combined with #3175. But I don't know about Eclipse's dependency rules and how they work. |
|
I'd like to propose reworking this PR to use a workspace-only configuration instead of the current instruction-based approach. Here's the plan: RationaleThe current design exposes
Proposed Changes
This keeps the feature working in Eclipse immediately and sets up the infrastructure for Maven/Gradle plugins to use the same API in the future. Impact
|
6e67b78 to
4767395
Compare
|
In 4767395 I implement my earlier comment #7209 (comment) The rebuildtrigger policy is no longer an instruction but now a setting on the Workspace.java object which can be set by build tools. I updated the PR description accordingly. |
Actually, from what you've described below, this PR is trying to do something better than merely "timestamp of the jar", because it is calculating a digest of the public API. This is good, because there are plenty of cases where the timestamp of the jar may change but the public API does not (eg, internal code only, or
Ok, I had a closer look. Forgive me if I have missed anything, but... It works by "cheating" and rewinding the clock on the Jar update if it detects that the API wasn't changed by the update. This seems to me a little bit hacky. In effect, you're lying to downstream tools about whether or not the jar has actually changed. Not every downstream dependency of a Jar is another Java build. What happens if you have a final build step that creates a "fat jar" from the dependent jars? Or if you have a "deploy" phase in your build? They may not run as they will conclude that there is no need as there have been no upstream changes. Another potential problem: If you have a jar dependency on the path that you are pulling classes into your bundle via (eg) A less important issue: there is also a small window where the timestamp on disk will be the "real" value before it gets rewound to the "old" value, which could lead to race conditions if the build is multi-threaded. The advantage of this approach vs #3175 is its portability - it can assist in other build drivers. However, it may also cause issues because downstream builds expect the timestamp to update, and not every downstream dependency is a Java build (eg, "deploy"). However, I think that #3175 is conceptually cleaner. Having said all of that, given the pain of cascading builds, it might be worth all of the complications this change could introduce in some circumstances, especially if you can disable it. |
|
Thanks @kriegfrj for your input.
Exactly. Yes I cheat with the timestamp as a vehicle to avoid downstream rebuilts.
It is :) But maybe this can help us: Now we know the exact place where the timestamp comparison happens to do that "has the bundle changed so that I have to rebuild" decision. bnd/bndtools.builder/src/org/bndtools/builder/DeltaWrapper.java Lines 228 to 240 in 2fd821a I wonder if we could find a way to replace or extend the timestamp check with another kind of "check" so that we must not "cheat" with the timestamp. We could reuse all the other things which are now done in this PR (calculating a digest of the public API and content). I will comment on the other issues:
hmm... valid cases. I wonder what would be the required check (in words) to build that bundle anyway? Something like: My current timestamp "cheat" is just about the upstream bundle "knowing" that its API and content have not changed and the timestamp is currently my "hack" to transport that knowledge to consumers (since this check is already there and used). But we know that the timestamp alone is not enough information and also cheating. To summarize: there are two parties having to make a decision:
true.
Yes this is something I value, since Eclipse IDE usage is declining and it would be great to find something which could help other tooling in the future too - although I myself and our company are using Eclipse IDE and those long builds are our main pain I try to address here.
I need to read into #3175 more.
Yes. That is why I have made that property UPDATE: I noticed that bnd/bndtools.builder/src/org/bndtools/builder/BndtoolsBuilder.java Lines 183 to 186 in 2fd821a |
👍 Thank you for creating something for me to comment on!
😆
We definitely can; however the
This is a good summary. I think for build dependencies to work properly (ideally), it should be the upstream builders' job to advertise its current state accurately, and the downstream builders' job to decide whether or not it needs to rebuild. In the general case, only the downstream builder can make the decision because only it knows what it is trying to build.
The thing is that this optimisation is most helpful for incremental building, not so much for CI. Another way I thought of that could work and would be portable is for the driver to explode the Jar and then have the depend on the classes rather than on the Jar. I didn't look at this in more detail as it seemed more complicated than the approach I was looking at. Another thought: I didn't look into it in detail, but from memory Gradle does have sophisticated enough dependency checking mechanism that you could achieve the same fine-grained dependency checking as Eclipse can do. You could store the API hash in the dependency object.
Me too...
This is true, but |
Definitly: This was one reason I decided against a instruction and just made it a programmatic option at the Workspace.
I just spent a couple of hours learning the above the hard way. But in the end JavaBuilder
I give up for now. I will think about it a bit more, and maybe I have a better idea in the future. |
f27ea1c to
639d7fe
Compare
|
Update:
|
639d7fe to
26dee45
Compare
There was a problem hiding this comment.
Pull request overview
This PR introduces a new core JarLifecycleListener plugin extension point in bndlib to hook into the JAR write lifecycle, and implements an Eclipse Bndtools “rebuild trigger policy” on top of it to avoid unnecessary downstream rebuild cascades by preserving output JAR timestamps when content/API is unchanged.
Changes:
- Add
aQute.bnd.service.JarLifecycleListenerand invoke it fromProject.saveBuildWithoutClose()(before/after JAR write), bumping theaQute.bnd.servicepackage version. - Add Bndtools rebuild trigger policy implementation (digest sidecar files + timestamp preservation) plus preference UI and toolbar indicator.
- Add/adjust docs and introduce tests/testdata for the new lifecycle hook and rebuild policy behavior.
Reviewed changes
Copilot reviewed 23 out of 24 changed files in this pull request and generated 16 comments.
Show a summary per file
| File | Description |
|---|---|
| docs/_plugins/jar-lifecycle.md | New docs page describing the JarLifecycleListener plugin interface. |
| docs/_plugins/bndtools-rebuild-policy-plugin.md | New docs page describing the Bndtools rebuild trigger policy plugin behavior. |
| docs/_plugins/00-overview.md | Plugin docs overview adjusted; adds registration/error-handling guidance and repositions repository tagging section. |
| docs/_chapters/150-build.md | Minor formatting/indentation fixes in build chapter. |
| bndtools.core/testdata/ws/p-stale/bnd.bnd | New test workspace project for rebuild trigger tests. |
| bndtools.core/testdata/ws/p-stale/.gitignore | Ignores generated output for test workspace project. |
| bndtools.core/testdata/ws/p-stale-dep/bnd.bnd | New dependent test project definition. |
| bndtools.core/testdata/ws/p-stale-dep/.gitignore | Ignores generated output for dependent test project. |
| bndtools.core/testdata/ws/cnf/build.bnd | New cnf build config for the test workspace. |
| bndtools.core/test/bndtools/tasks/RebuildTriggerTest.java | New tests validating digest sidecar behavior and timestamp preservation semantics. |
| bndtools.core/src/bndtools/preferences/ui/BndPreferencePage.java | Adds build preferences page ID constant. |
| bndtools.core/src/bndtools/preferences/ui/BndBuildPreferencePage.java | Adds rebuild trigger policy radio-button UI and persists preference. |
| bndtools.core/src/bndtools/preferences/BndPreferences.java | Stores new rebuild trigger policy preference with defaults and accessors. |
| bndtools.core/src/bndtools/explorer/BndtoolsExplorer.java | Adds toolbar indicator for rebuild trigger policy and opens preferences on click. |
| bndtools.core/src/bndtools/central/RebuildTriggerPolicyPlugin.java | New JarLifecycleListener implementation wiring rebuild policy into the JAR lifecycle. |
| bndtools.core/src/bndtools/central/RebuildTriggerPolicy.java | New digest/timestamp preservation policy implementation (content + API digest). |
| bndtools.core/src/bndtools/central/Central.java | Applies rebuild trigger policy preference to workspaces and re-applies after refresh. |
| bndtools.core/bnd.bnd | Adds biz.aQute.bnd.test to the testpath. |
| bndtools.builder/src/org/bndtools/builder/CnfWatcher.java | Re-applies rebuild policy after workspace refresh. |
| bndtools.builder/src/org/bndtools/builder/BndtoolsBuilder.java | Re-applies rebuild policy after cnf-triggered refresh. |
| biz.aQute.bndlib/src/aQute/bnd/service/package-info.java | Bumps package version due to new public service interface. |
| biz.aQute.bndlib/src/aQute/bnd/service/JarLifecycleListener.java | New core plugin interface for before/after JAR write hooks. |
| biz.aQute.bndlib/src/aQute/bnd/build/Project.java | Calls JarLifecycleListener hooks around JAR write. |
| biz.aQute.bndlib.tests/test/test/ProjectTest.java | Adds tests asserting listener callbacks and context map behavior. |
Comments suppressed due to low confidence (6)
bndtools.core/test/bndtools/tasks/RebuildTriggerTest.java:180
- This block says it "writes a fake API digest sidecar", but no file is written and the variable is unused. The comments should match what the test actually does to avoid future confusion.
// Write a fake API digest sidecar. The next build will produce
// the same resource-only bundle (no Export-Package), so
// calcApiDigest() returns null. However, when we change the
// project content the content digest will differ. To simulate
// the API-digest-unchanged path, we need a non-null API digest.
// We'll change the content, then verify the mechanism by
// checking the timestamp. Since calcApiDigest returns null for
// resource-only bundles, the API digest path won't engage here,
// but we can verify the infrastructure is correctly wired:
bndtools.core/test/bndtools/tasks/RebuildTriggerTest.java:199
- newDigest is computed but never used. Either assert on it (e.g., changed from previous digest) or remove the unused variable.
File jarFile2 = secondBuild[0];
// Content changed so the digest should differ
String newDigest = IO.collect(digestFile).trim();
// The JAR should have a new timestamp since both content and
biz.aQute.bndlib/src/aQute/bnd/build/Project.java:2166
- The debug log message refers to "JarArtifactLifecycleListener.afterJarWrite", but the actual callback is JarLifecycleListener.afterWrite. Aligning the message with the real interface improves diagnosability.
try {
listener.afterWrite(project, outputFile, jar, context);
} catch (Exception e) {
logger.debug("Exception in JarArtifactLifecycleListener.afterJarWrite", e);
}
biz.aQute.bndlib/src/aQute/bnd/service/JarLifecycleListener.java:160
- Several Javadoc link labels/reference names still mention beforeJarWrite/afterJarWrite even though the API uses beforeWrite/afterWrite. This makes the rendered docs inconsistent and harder to follow.
* The listener can store computed values in the {@code context} map for use
* in the {@link #afterWrite afterJarWrite} phase.
* <p>
*
* @param project the project being built
* @param jar the in-memory JAR object, before being written to disk
* @param outputFile the file where the JAR will be written
* @param context a mutable map for sharing data between before and after
* phases; persists across both phases for a single JAR write
* @throws Exception if an error occurs; the exception is logged but does
* not prevent other listeners or the JAR write from proceeding
*/
default void beforeWrite(Project project, Jar jar, File outputFile, Map<String, Object> context)
throws Exception {}
/**
* Invoked after the JAR has been successfully written to the output file.
* <p>
* This phase is suitable for:
* <ul>
* <li>Storing metadata (digests, signatures) in sidecar files.</li>
* <li>Modifying file properties (timestamps, permissions).</li>
* <li>Triggering post-write notifications or side-effects.</li>
* <li>Validating the written file.</li>
* </ul>
* <p>
* The listener can retrieve data computed in the {@link #beforeWrite
* beforeJarWrite} phase from the {@code context} map.
* <p>
bndtools.core/test/bndtools/tasks/RebuildTriggerTest.java:184
- apiDigestFile is declared but never used (and no API digest is written), which adds confusion without asserting anything. Remove the unused variable or actually write/assert against the sidecar file in a test that can compute an API digest.
File apiDigestFile = new File(jarFile.getParentFile(), jarFile.getName() + ".api-digest");
biz.aQute.bndlib/src/aQute/bnd/service/JarLifecycleListener.java:173
- The afterWrite() Javadoc still refers to "beforeJarWrite" in link labels/text, but the API method is named beforeWrite. This makes generated Javadoc inconsistent.
* The listener can retrieve data computed in the {@link #beforeWrite
* beforeJarWrite} phase from the {@code context} map.
* <p>
* <b>Note:</b> The JAR object may be in a different state after writing
* (e.g., resources may have been closed). Listeners should not rely on the
* JAR being fully functional; use the {@code outputFile} and the
* {@code context} map instead.
* <p>
*
* @param project the project that was built
* @param outputFile the file to which the JAR was written; the file is
* guaranteed to exist on the filesystem
* @param jar the JAR object that was written (state may have changed)
* @param context the same mutable map passed to {@link #beforeWrite
* beforeJarWrite}, allowing access to data computed in that
* phase
5fa0a3b to
bd9ac00
Compare
After building a project's JAR, compute a timeless content digest and compare with the previously stored digest. If the content is unchanged, preserve the old JAR's timestamp. This prevents dependent projects from seeing a newer timestamp and triggering unnecessary rebuilds. The optimization uses Jar.getTimelessDigest() which ignores build-time-specific data (BND_LASTMODIFIED, version qualifier) to determine if the meaningful content has changed. A .digest sidecar file stores the hex digest alongside each build output. Agent-Logs-Url: https://github.com/bndtools/bnd/sessions/662b413c-3754-4104-a50f-cb8503721220 Signed-off-by: Christoph Rueger <chrisrueger@gmail.com> Add Eclipse preference UI for rebuild trigger policy setting Adds a new 'Rebuild Trigger Policy' dropdown to the Bnd Build preferences page, letting users choose between 'Always rebuild' and 'API-based (skip if API unchanged)' without editing cnf/build.bnd. Signed-off-by: Christoph Rueger <chrisrueger@gmail.com> Add rebuild trigger policy indicator to explorer toolbar Adds a visual indicator in the Bndtools Explorer toolbar displaying the current rebuild trigger policy. The indicator updates when preferences change and opens the build preferences page when clicked. Signed-off-by: Christoph Rueger <chrisrueger@gmail.com> Make rebuild trigger policy workspace-level, not project-scoped Refactor rebuild trigger policy from a project-level build.bnd property to a workspace-level programmatic setting. This allows build tools like Eclipse IDE to control the policy for the entire workspace via the new Workspace.setRebuildTriggerPolicy() API rather than users configuring it in build.bnd. Simplify Eclipse preferences by removing the 'Default' option—the workspace now always has a policy value (defaults to 'always'). Move documentation from instruction file to build chapter and update for programmatic-only configuration. Bump package version to 4.8.0. Signed-off-by: Christoph Rueger <chrisrueger@gmail.com> Refactor rebuild policy into lifecycle plugin Introduces a new `JarLifecycleListener` service hook in bndlib and wires `Project` to call listeners before/after JAR writes, replacing the in-core rebuild trigger policy logic. The old `Workspace` rebuild policy state and related `Constants` entries are removed, and the service package version is bumped to 4.11.0. Bndtools now owns rebuild-trigger behavior via `RebuildTriggerPolicy` and `RebuildTriggerPolicyPlugin`, which are registered from `Central.applyRebuildTriggerPolicy()` based on preferences. Preference/UI/explorer code is updated to use the new policy constants, and tests that depended on `Workspace#setRebuildTriggerPolicy` are temporarily commented with TODO markers. Add rebuild trigger policy plugin tests Move content-hash/API-digest rebuild behavior tests out of `ProjectTest` into a new `bndtools.core` test suite that exercises `RebuildTriggerPolicyPlugin` with dedicated workspace testdata. Add `ProjectTest` coverage for `JarLifecycleListener` integration, verifying `beforeWrite`/`afterWrite` invocation and shared context data flow. Also wire `biz.aQute.bnd.test` into `bndtools.core` testpath and remove stale commented cleanup lines. Signed-off-by: Christoph Rueger <chrisrueger@gmail.com> Document JAR lifecycle and rebuild policy plugins Add new plugin documentation for `JarLifecycleListener` and the Bndtools rebuild trigger policy, including lifecycle phases, context usage, registration, sidecar digests, and policy behavior (`always` vs `api`). Update the plugin overview with generic registration/error-handling guidance and retain repository tagging details in the index page. Signed-off-by: Christoph Rueger <chrisrueger@gmail.com> Co-Authored-By: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-Authored-By: chrisrueger <188422+chrisrueger@users.noreply.github.com>
bd9ac00 to
3f3a2cf
Compare

This pull request introduces a plugin-based architecture for intercepting the JAR artifact lifecycle, enabling build optimizations like rebuild trigger policies without modifying core bnd code.
Overview
The core contribution is a new
JarLifecycleListenerplugin interface that allows tools to observe and participate in the JAR write lifecycle (before and after a JAR is written to disk). This enables:How It Works
1. Core Plugin Interface
A new
JarLifecycleListenerinterface in the core allows plugins to hook into the JAR write lifecycle:Listeners receive a shared context map that persists between before/after phases, enabling stateless, thread-safe data exchange without storing state in the listener itself.
2. Bndtools Plugin Implementation
The rebuild trigger policy optimization (previously hardcoded in
Project.java) is now implemented as an optional bndtools plugin that:.digest,.api-digest) and restores the old timestamp if content is unchangedThis keeps the optimization completely out of core bnd and makes it a bndtools-specific feature.
3. Workspace-Level Configuration
The rebuild trigger policy is configured at the workspace level via:
Bndtools Eclipse IDE reads user preferences and applies the policy before building.
Rebuild Trigger Policies
Two policies are supported:
always(default)apiFiles Changed
Core changes (minimal, plugin-focused):
Project.java– AddedJarLifecycleListenerplugin calls insaveBuildWithoutClose()and artifact cleanup hooks inbuildLocal()Workspace.java– AddedgetRebuildTriggerPolicy()andsetRebuildTriggerPolicy()methodsBndtools changes (implements the optimization):
RebuildTriggerPolicyListener.java– Plugin implementation for rebuild optimizationBndPreferences.java– Eclipse preferences for rebuild trigger policyBndBuildPreferencePage.java– UI for rebuild policy selectionBndtoolsExplorer.java– Toolbar indicator showing current rebuild policyTests:
JarLifecycleListenerplugin interface works correctlyBenefits
Eclipse Integration
Users can enable the optimization via:
Preferences → Bndtools → Build → Rebuild Trigger Policy
The toolbar of Bndtools Explorer displays the current status:
Also show the current status in the toolbar of Bndtools Explorer:
Future Extensibility
This design allows other tools to adopt similar optimizations:
pom.xml, register plugin, callworkspace.setRebuildTriggerPolicy()build.gradle, register pluginEach tool controls when and how the plugin is applied; the core mechanism remains unified.
Limitations
lastModified()value. This works well in most incremental build scenarios but may not perfectly reflect semantic changes in all edge cases.In practice, this conservative approach still provides substantial benefits by preventing most unnecessary rebuilds while maintaining simplicity and reliability.